Skip to content

test(ci): isolate semantic probe fixture coverage - #5375

Open
Duang777 wants to merge 9 commits into
mainfrom
codex/fix-coverage-temp-source
Open

Duang777 wants to merge 9 commits into
mainfrom
codex/fix-coverage-temp-source

Conversation

@Duang777

Copy link
Copy Markdown
Collaborator

Summary

Prevent the semantic probe CLI fixture from inheriting pytest and coverage startup variables. The fixture runs from a temporary synthetic checkout; tracing that child records temporary loopx/a.py paths in shard coverage artifacts, which do not exist on the separate aggregation runner.

This keeps the real CLI assertions unchanged and excludes only the synthetic child process from the parent coverage session. It follows the existing CI fixture isolation pattern from dd17cc259. No production code or coverage threshold changes.

Failure evidence

On #5367 exact head f5995f102, all four Python test shards passed, then the required pytest aggregation job failed while combining coverage:

No source for code: /tmp/pytest-of-runner/.../test_cli_inspects_disjoint_ind0/loopx/a.py

Shard 1 and shard 2 reports both contained temporary loopx/a.py entries. The tests were introduced on main by #5317; their earlier CI run never reached coverage combination because other shard failures stopped the aggregate first.

Validation

  • Before the fix, a local cross-runner simulation passed both CLI tests, deleted the pytest temp root, then reproduced coverage report exit 1 with the same missing-source error.
  • After the fix, the same simulation passes after temp-root deletion.
  • Full semantic probe suite: 17 passed.
  • Python CI workflow contract suite: 290 passed.
  • Ruff and git diff --check: passed.
  • loopx canary premerge --from-git-diff: passed, 2 selected canaries, 0 failures, 0 manual holds.
  • Bits Code Guard full-file review: no P0-P2 findings.

Maintainer review and merge are required.

Signed-off-by: duanjialing.777 <duanjialing.777@bytedance.com>
@Duang777

Copy link
Copy Markdown
Collaborator Author

The remaining typescript-core (1/3) failure is confirmed on main run 36770551543 and comes from the duplicate bare SHA-256 matcher introduced by #5364, not from this PR. A minimal main-based fix is now in #5377; this branch remains unchanged at exact head 086f2643355d36b62ccf74cbdac6200ca31a498f.

@Duang777

Copy link
Copy Markdown
Collaborator Author

A second inherited failure appeared in test-shard (2): test_runtime_source_churn_has_a_stable_readiness_diagnostic observed ready instead of package_invalid. The current main fixture deletes later.ts from one parallel read without synchronizing the other read; if later.ts is already prefetched, the expected missing-file retry does not occur. #5367 makes this race deterministic with an Event and its exact-head shard 2 passed. This is independent of the sole coverage-helper change in #5375, so no code change is planned here.

@Duang777

Copy link
Copy Markdown
Collaborator Author

The newly completed test-shard (4) failure is the second assertion covered by #5367: test_runtime_request_source_churn_raises_a_stable_startup_diagnostic observed runtime_exited_before_ready instead of packaged_runtime_source_unstable. #5367 synchronizes this source-churn fixture and its exact-head test-shard (4) passed. This is independent of #5375’s sole coverage-helper change, so this branch remains unchanged.

@Duang777

Copy link
Copy Markdown
Collaborator Author

The final test-shard (3) failure is another inherited main regression: #5366 advanced the usage notice to version 5 but left test_source_first_usage_disclosure_keeps_json_pure_and_does_not_send pinned to 4. I reproduced the exact 5 == 4 assertion on clean origin/main@9c9b90d8e; the one-line test correction is isolated in #5379. No #5375 code change is needed.

Signed-off-by: duanjialing.777 <duanjialing.777@bytedance.com>
@Duang777

Duang777 commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator Author

Exact-head update for 254d66b29341382c4d4c3b94382806e0bb59ae93:

  • Synced with repaired main@638d0e39cff09948df1816a4a5c5ccce70262c57 through a conflict-free signed-off merge commit.
  • The PR-owned diff remains limited to 9 added test lines in tests/architecture/test_semantic_development_probe.py.
  • Focused Python validation passed: 74/74 across the semantic probe, source CLI entrypoint, and Turn journal runtime-readiness suites.
  • npm run typecheck:control-plane passed.
  • tests/control_plane_ts/content_digest_single_owner.test.ts passed: 17/17.
  • examples/control_plane/cli-output-budget-regression-smoke.py passed against current origin/main.
  • git diff --check passed.

The previous run was based on main@64402b4a8, whose duplicate BARE_SHA256_PATTERN import made the TypeScript runtime exit before ready. Shards 1, 2, and 4 failed from that unavailable runtime, and shard 3 was cancelled after the run was already red. #5391 repaired and landed that main regression.

Fresh exact-head CI run 36818756640 is complete: 24 jobs passed, 2 optional jobs were skipped, and no jobs failed. No #5375-owned failure was found, and no merge action was taken.

…mp-source

Signed-off-by: duanjialing.777 <duanjialing.777@bytedance.com>
…mp-source

Signed-off-by: duanjialing.777 <duanjialing.777@bytedance.com>
Signed-off-by: duanjialing.777 <duanjialing.777@bytedance.com>
…mp-source

Signed-off-by: duanjialing.777 <duanjialing.777@bytedance.com>
…mp-source

Signed-off-by: duanjialing.777 <duanjialing.777@bytedance.com>
@Duang777

Duang777 commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator Author

Exact-head update for aa1ef4299827a1bd01fc508150bd38704e90d269:

  • Synced through signed-off, conflict-free merge commits with current main@3ad3269af4d2cfae2085693823dcfcda522f9a2b.
  • Focused validation passed: 74 Python tests, 17 digest-owner TypeScript tests, control-plane TypeScript typecheck, Ruff, and git diff --check.
  • After test(host): publish process markers atomically #5365 entered main during validation, the affected Host/Codex CLI suite also passed: 48/48.

Exact-head CI run 36825867638 is complete: 24 jobs passed, 2 optional jobs were skipped, and no jobs failed. This includes all four Python shards, combined coverage, and merge-gate. The PR is ready for maintainer review; no merge action was taken.

…mp-source

Signed-off-by: duanjialing.777 <duanjialing.777@bytedance.com>
…mp-source

Signed-off-by: duanjialing.777 <duanjialing.777@bytedance.com>
@Duang777

Duang777 commented Oct 1, 2026

Copy link
Copy Markdown
Collaborator Author

Exact-head update for 3d2e9862bb93aa27fc9c349c47ab03b2c5dbc923 on main@060e4698a39264988391b3f11bc18c91eb81d27f:

  • Synced through signed-off, conflict-free merge commits.
  • The PR-owned diff remains limited to tests/architecture/test_semantic_development_probe.py.
  • Focused validation passed: 17 Python tests, Ruff, and git diff --check.
  • The previous exact-head CI run completed with no failures. Fresh CI is running on this head.

The PR remains ready for maintainer review. No merge action was taken.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant